In conventional systems, data compression or source coding is the process of encoding information using fewer bits than an uncompressed representation would use through use of specific encoding schemes. Compression is useful because it helps reduce the consumption of expensive resources, such as storage space or transmission bandwidth. On the downside, compressed data must be decompressed to be used, and this extra processing may be detrimental to some applications. This difficulty is especially true given that most compression or encoding algorithms today compress data at the bit level.
For example, conventional lossless compression algorithms seek to leverage statistical redundancies extant in the bit representations (e.g., 1's and 0's) that compose various types of data. Lossless compression schemes are reversible so that the original data can be reconstructed exactly as it existed prior to encoding. An example of lossless compression is ZIP file format, which is often employed for compressing documents or other data entities or data formats that require accurate recreation at the bit level. On the other hand, lossy encoding can provide greater compression ratios at the expense of some data loss. Generally, lossy compression is employed in connection with images, audio, or video files, sacrificing some degree of fidelity to attain a higher level of compression. Examples of lossy compression are MP3, MPEG, JPG formats. Both forms of compression have relatively rigid maximum compression ratios.
In the domain of computer network systems, bit-level compression of data can reduce bandwidth utilization, which can lessen the payload of traffic through potential resource bottlenecks. The trade-off is an increase in the required computational power to perform the compression/decompression algorithms, which commensurately increases the costs associated with various network equipment. In many instances, the cost-benefit analysis of this increased expense is not justified, especially when the occurrence of a resource bottleneck is a relatively low-probability event. For example, consider a network monitoring system (NMS) that tracks alarm messages from a network or a particular portion of a network.
Typically, the NMS will received alarm messages from network equipment (e.g., access multiplexers) when a fault condition arises with or is detected by the equipment. These fault conditions are often isolated events, and do not overly burden the NMS. However, in certain situations, “alarm storms” can arise. Alarm storms typically occur when many alarm messages are generated in a relatively short period of time, often due to a common or related set of events, e.g. due to malicious denial of service attacks or even due to natural forces such as weather. During alarm storms or other times of high alarm message input to a gateway of a NMS, the NMS is conventionally forced to shut down or discard much information that could be helpful to an upstream analysis system that might receive alarms from multiple NMS's to determine the nature of the problem. Accordingly, it would be beneficial to employ compression techniques to lessen the strain on the NMS during such times, yet bit-level compression techniques, with their limited compression ratios and higher additional costs do not appear adequate to address these or other suitable situations.